Add CUDA version compatibility check #1412

Andy-Jost · 2025-12-18T19:06:15Z

Summary

Adds a version compatibility check that warns users when cuda-bindings was compiled against a newer CUDA major version than the installed driver supports.

Changes

cuda-bindings

Added check_cuda_version_compatibility() function in cuda/bindings/utils/_version_check.py
Compares compile-time CUDA_VERSION vs runtime cuDriverGetVersion()
Exported from cuda.bindings.utils
Added comprehensive unit tests in tests/test_version_check.py

cuda-core

Device.__new__ calls check_cuda_version_compatibility() after cuInit succeeds
Imports the function directly from cuda.bindings.utils

Rationale

When cuda-bindings is built against CUDA 13 headers but the user's driver only supports CUDA 12, many features will silently fail or behave unexpectedly. This check provides early, clear feedback:

UserWarning: cuda-bindings was built against CUDA 13.0, but the installed driver 
only supports CUDA 12.8. Some features may not work correctly. Consider updating 
your NVIDIA driver. Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to suppress this warning.

Design

Provided by cuda-bindings: The version check implementation lives in cuda.bindings.utils since it checks cuda-bindings' compile-time version
Invoked by cuda-core: Called when Device first triggers CUDA initialization
Runs once: Uses a module-level flag to ensure the check runs only once per process
Non-blocking: Warning only, does not prevent operation
Suppressible: Set CUDA_PYTHON_DISABLE_VERSION_CHECK=1 to disable
Major version only: No warning for minor version differences (handled by graceful degradation per PR Handle unsupported device attributes gracefully #1409)

Future Work

We could not find a suitable place to invoke the version check automatically within cuda-bindings itself (e.g., hooking into cuInit), so the check is currently triggered by cuda-core. This may be revisited in the future.

Test Coverage

7 tests in cuda-bindings covering:

No warning when driver is newer
No warning when same major version
Warning when compile major > driver major
Warning only issued once per process
Suppression via environment variable
Silent handling of driver errors
Silent handling of missing CUDA_VERSION attribute

Related Work

Complements PR Handle unsupported device attributes gracefully #1409 which handles graceful degradation for minor version mismatches

copy-pr-bot · 2025-12-18T19:06:18Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Andy-Jost · 2025-12-18T19:06:28Z

/ok to test 7ce325c

Andy-Jost · 2025-12-18T19:08:17Z

/ok to test 1962e35

github-actions · 2025-12-18T19:20:22Z

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1412/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1412/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

cuda_core/cuda/core/_utils/cuda_utils.pyx

leofang · 2025-12-18T20:28:01Z

cuda_bindings/docs/source/environment_variables.rst


 - ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

+- ``CUDA_PYTHON_DISABLE_VERSION_CHECK`` : When set to 1, suppresses the warning that is issued when ``cuda.core`` detects that ``cuda-bindings`` was compiled against a newer CUDA major version than the installed driver supports. This warning helps identify version mismatches that may cause features to not work correctly.


This env var is added to cuda-bindings docs, but the actual change is made to cuda-core? Can we move them to the right place?

Actually, thinking about this we probably should do this check in cuda-bindings instead of cuda-core, since it's what actually matters. The question is if there is a good place that would allow lazy-init. Remember, both cuda-core and cuda-bindings can be installed and imported on CPU-only machines, without driver or a physical GPU installed. I can't think of an obvious solution off top of my head.

What about cuInit? That seems like a logical place for the version check, though the change might be more complex.

As discussed, cuda-bindings provides the check; cuda-core invokes it. Other consumers of cuda-bindings can also invoke the check.

cuda_core/cuda/core/_utils/cuda_utils.pyx

Andy-Jost · 2025-12-19T00:10:25Z

/ok to test 73e5a43

Andy-Jost · 2025-12-19T00:48:39Z

/ok to test ddd92bd

cuda_bindings/cuda/bindings/utils/_version_check.py

rparolin

We should issue a warning to the user if we can't fetch the driver version.

cuda_bindings/cuda/bindings/utils/_version_check.py

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

# Conflicts: # cuda_core/pixi.lock

Import warn_if_cuda_major_version_mismatch locally in Device.__new__ after cuInit, using try/except/else pattern instead of module-level import with lambda fallback.

Andy-Jost · 2026-01-08T18:00:34Z

/ok to test b2083ed

Extract Device.__new__ logic into cdef helper functions: - Device_ensure_cuda_initialized(): cuInit + version check - Device_resolve_device_id(): resolve None to current device or 0 - Device_ensure_tls_devices(): create thread-local singletons Reduces Device.__new__ from ~60 lines to ~12 lines. Helpers placed after Device class following memory module pattern.

Andy-Jost · 2026-01-08T18:19:03Z

/ok to test fdb3a7e

cuda_bindings/tests/test_version_check.py

Andy-Jost · 2026-01-09T21:30:52Z

/ok to test acadb31

Integrate RAII handles architecture from main while preserving the version compatibility check feature. Resolved conflict in _device.pyx by keeping the refactored helper functions and updating field names to match RAII object structure.

Andy-Jost · 2026-01-13T19:42:51Z

/ok to test 3a5c210

Andy-Jost · 2026-01-14T01:12:48Z

/ok to test 424a113

Andy-Jost · 2026-01-14T01:42:35Z

/ok to test 6071609

leofang

Sorry for my late review.

cuda_bindings/cuda/bindings/utils/_version_check.py

leofang · 2026-01-14T03:20:20Z

cuda_bindings/docs/source/environment_variables.rst


 - ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

+- ``CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING`` : When set to 1, suppresses warnings about CUDA major version mismatches between ``cuda-bindings`` and the installed driver.


TBH I am not sure if this env var is really useful, because the added API is entirely opt-in?

The version check can be triggered indirectly from the end user's POV. For example, by using cuda-core. Without this, each library that calls the version-check function would need its own opt-out, and users might need to set them all. To me it makes sense to place the opt-out alongside the implementation and there is little/no cost.

I insist that this is useless because this is listed in the cuda-bindings docs, not cuda-core. But cuda-bindings does not call this function anywhere, whereas cuda-core users don't necessarily look at cuda-bindings docs. Can we please create a new environment_variables.rst for cuda-core if we think this is helpful there?

xref: #1412 (comment)

Even better: Document this env var in both docs.

cuda_bindings/tests/test_version_check.py

cuda_core/cuda/core/_device.pyx

Andy-Jost · 2026-01-14T17:49:45Z

/ok to test 96532b2

Andy-Jost · 2026-01-14T17:51:15Z

/ok to test 973af9b

Replace setup_method/teardown_method with a pytest fixture that uses monkeypatch to properly save and restore the original value of _major_version_compatibility_checked after each test. Minor change to Cython cdef inline helper function signature.

Andy-Jost · 2026-01-14T18:03:55Z

/ok to test 7f23b08

cuda_core/cuda/core/_device.pyx

Andy-Jost added this to the cuda.core beta 11 milestone Dec 18, 2025

Andy-Jost added P0 High priority - Must do! feature New feature or request cuda.core Everything related to the cuda.core module labels Dec 18, 2025

Andy-Jost self-assigned this Dec 18, 2025

Andy-Jost requested review from leofang, rparolin and rwgk December 18, 2025 19:06

Andy-Jost force-pushed the runtime-version-check branch from 7ce325c to 1962e35 Compare December 18, 2025 19:08

mdboom approved these changes Dec 18, 2025

View reviewed changes

Andy-Jost mentioned this pull request Dec 18, 2025

fix: derive CUDA major version from headers for build #1395

Merged

2 tasks

leofang requested changes Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

leofang reviewed Dec 18, 2025

View reviewed changes

cuda_core/cuda/core/_utils/cuda_utils.pyx Outdated Show resolved Hide resolved

Andy-Jost force-pushed the runtime-version-check branch 2 times, most recently from 8a0d248 to 73e5a43 Compare December 19, 2025 00:09

Andy-Jost force-pushed the runtime-version-check branch from 73e5a43 to ddd92bd Compare December 19, 2025 00:47

leofang self-requested a review December 19, 2025 01:22

rparolin reviewed Dec 19, 2025

View reviewed changes

cuda_bindings/cuda/bindings/utils/_version_check.py Outdated Show resolved Hide resolved

rparolin reviewed Dec 19, 2025

View reviewed changes

cuda_bindings/cuda/bindings/utils/_version_check.py Outdated Show resolved Hide resolved

rparolin requested changes Dec 20, 2025

View reviewed changes

rwgk reviewed Dec 26, 2025

View reviewed changes

Andy-Jost added 2 commits January 7, 2026 13:09

Add CUDA major version compatibility check

d999f40

Add warn_if_cuda_major_version_mismatch() to cuda-bindings that warns when cuda-bindings was compiled for a newer CUDA major version than the installed driver supports. Called by cuda.core on first Device access.

Merge remote-tracking branch 'origin/main' into runtime-version-check

62dfcca

# Conflicts: # cuda_core/pixi.lock

Andy-Jost requested a review from rparolin January 8, 2026 17:48

Andy-Jost added 2 commits January 8, 2026 09:59

Move version check import to local scope

73611ed

Import warn_if_cuda_major_version_mismatch locally in Device.__new__ after cuInit, using try/except/else pattern instead of module-level import with lambda fallback.

Merge remote-tracking branch 'origin/main' into runtime-version-check

b2083ed

Andy-Jost force-pushed the runtime-version-check branch from 07bf4a4 to b2083ed Compare January 8, 2026 18:00

Andy-Jost force-pushed the runtime-version-check branch from a92ed3b to fdb3a7e Compare January 8, 2026 18:18

rparolin reviewed Jan 9, 2026

View reviewed changes

cuda_bindings/tests/test_version_check.py Outdated Show resolved Hide resolved

rparolin approved these changes Jan 9, 2026

View reviewed changes

kkraus14 approved these changes Jan 13, 2026

View reviewed changes

Merge branch 'main' into runtime-version-check

3a5c210

Integrate RAII handles architecture from main while preserving the version compatibility check feature. Resolved conflict in _device.pyx by keeping the refactored helper functions and updating field names to match RAII object structure.

Andy-Jost force-pushed the runtime-version-check branch from acadb31 to 3a5c210 Compare January 13, 2026 19:42

Merge branch 'main' into runtime-version-check

424a113

Merge branch 'main' into runtime-version-check

6071609

rwgk approved these changes Jan 14, 2026

View reviewed changes

leofang requested changes Jan 14, 2026

View reviewed changes

Merge branch 'main' into runtime-version-check

7925693

Andy-Jost force-pushed the runtime-version-check branch from 96532b2 to 973af9b Compare January 14, 2026 17:51

Andy-Jost force-pushed the runtime-version-check branch from 973af9b to 27732f2 Compare January 14, 2026 17:56

Andy-Jost force-pushed the runtime-version-check branch from 27732f2 to 7f23b08 Compare January 14, 2026 18:03

rwgk approved these changes Jan 15, 2026

View reviewed changes

cuda_core/cuda/core/_device.pyx Show resolved Hide resolved


		- ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

		- ``CUDA_PYTHON_DISABLE_VERSION_CHECK`` : When set to 1, suppresses the warning that is issued when ``cuda.core`` detects that ``cuda-bindings`` was compiled against a newer CUDA major version than the installed driver supports. This warning helps identify version mismatches that may cause features to not work correctly.


		- ``CUDA_PYTHON_CUDA_PER_THREAD_DEFAULT_STREAM`` : When set to 1, the default stream is the per-thread default stream. When set to 0, the default stream is the legacy default stream. This defaults to 0, for the legacy default stream. See `Stream Synchronization Behavior <https://docs.nvidia.com/cuda/cuda-runtime-api/stream-sync-behavior.html>`_ for an explanation of the legacy and per-thread default streams.

		- ``CUDA_PYTHON_DISABLE_MAJOR_VERSION_WARNING`` : When set to 1, suppresses warnings about CUDA major version mismatches between ``cuda-bindings`` and the installed driver.

Add CUDA version compatibility check #1412

Are you sure you want to change the base?

Add CUDA version compatibility check #1412

Uh oh!

Conversation

Andy-Jost commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

cuda-bindings

cuda-core

Rationale

Design

Future Work

Test Coverage

Related Work

Uh oh!

copy-pr-bot bot commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

Andy-Jost commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Preview will be ready when the GitHub Pages deployment is complete.

Uh oh!

Uh oh!

leofang Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Andy-Jost commented Dec 19, 2025

Uh oh!

Uh oh!

Uh oh!

rparolin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Andy-Jost commented Jan 8, 2026

Uh oh!

Andy-Jost commented Jan 8, 2026

Uh oh!

Uh oh!

Andy-Jost commented Jan 9, 2026

Uh oh!

Andy-Jost commented Jan 13, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

Andy-Jost commented Jan 14, 2026

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

leofang Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Andy-Jost Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

leofang Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

leofang Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Andy-Jost commented Dec 18, 2025 •

edited

Loading

Andy-Jost Dec 18, 2025 •

edited

Loading